Detection of Spam Hosts and Spam Bots Using Network Flow Traffic Modeling

In this paper, we present an approach for detecting e-mail spam originating hosts, spam bots and their respective controllers based on network flow data and DNS metadata. Our approach consists of first establishing SMTP traffic models of legitimate vs. spammer SMTP clients and then classifying unknown SMTP clients with respect to their current SMTP traffic distance from these models. An entropy-based traffic component extraction algorithm is then applied to traffic flows of hosts identified as e-mail spammers to determine whether their traffic profiles indicate that they are engaged in other exploits. Spam hosts that are determined to be compromised are processed further to determine their command-and-control using a two-stage approach that involves the calculation of several flow-based metrics, such as distance to common control traffic models, periodicity, and recurrent behavior. DNS passive replication metadata are analyzed to provide additional evidence of abnormal use of DNS to access suspected controllers. We illustrate our approach with examples of detected controllers in large HTTP(S) botnets such as Cutwail, Ozdok and Zeus, using flow data collected from our backbone network.